feat: add multi-column support to `UpdateBy` `RollingFormula()` operator #6143

lbooker42 · 2024-09-26T21:02:52Z

No description provided.

malhotrashivam

Code review Part 1: All replicated files and files with smaller changes

Base/src/main/java/io/deephaven/base/ringbuffer/RingBuffer.java

...gine/table/impl/updateby/rollingformula/ringbuffervectorwrapper/RingBufferVectorWrapper.java

malhotrashivam · 2024-10-03T17:59:50Z

...gine/table/impl/updateby/rollingformula/ringbuffervectorwrapper/RingBufferVectorWrapper.java

+            // noinspection unchecked
+            result = new ObjectRingBufferVectorWrapper<T>((ObjectRingBuffer<T>) buffer, (Class<T>) componentType);


Please check to get full test coverage.

...a/io/deephaven/engine/table/impl/updateby/rollingminmax/ComparableRollingMinMaxOperator.java

...e/table/impl/updateby/rollingformulamulticolumn/windowconsumer/RingBufferWindowConsumer.java

...ble/impl/updateby/rollingformulamulticolumn/windowconsumer/CharRingBufferWindowConsumer.java

...e/impl/updateby/rollingformulamulticolumn/windowconsumer/ObjectRingBufferWindowConsumer.java

table-api/src/main/java/io/deephaven/api/updateby/UpdateByOperation.java

engine/table/src/test/java/io/deephaven/engine/table/impl/updateby/TestRollingFormula.java

engine/table/src/main/java/io/deephaven/engine/table/impl/updateby/UpdateByOperatorFactory.java

table-api/src/main/java/io/deephaven/api/updateby/spec/RollingFormulaSpec.java

.../engine/table/impl/updateby/rollingformulamulticolumn/RollingFormulaMultiColumnOperator.java

Base/src/main/java/io/deephaven/base/ringbuffer/RingBuffer.java

malhotrashivam · 2024-10-09T20:55:18Z

...va/io/deephaven/engine/table/impl/updateby/rollingformula/BooleanRollingFormulaOperator.java

-            formulaInputSource.set(new ObjectRingBufferVectorWrapper(windowValues, inputVectorType));
+            // noinspection rawtypes
+            formulaInputSource.set(new ObjectRingBufferVectorWrapper(windowValues, inputComponentType));


I didn't fully understand this change, can you please explain?
Also, can we use ByteRingBufferVectorWrapper here since we would have byte values internally? Please correct me if I am mistaken or if this is out of scope for this PR.

I didn't fully understand this change, can you please explain?

Fixing an earlier bug. I'm trying to track the fundamental component type of the input Object column to the Vector created from it.

Also, can we use ByteRingBufferVectorWrapper here since we would have byte values internally? Please correct me if I am mistaken or if this is out of scope for this PR.

This matches AggFormula in creating ObjectVector when assembling groups of Boolean input. The functions in Basic.java expect this input.

malhotrashivam · 2024-10-09T21:05:29Z

table-api/src/main/java/io/deephaven/api/updateby/spec/RollingFormulaSpec.java

+    public abstract String formula();
+


Does this change from List to a single value require any documentation update? Also, is this a breaking change?

Accepting multiple formula was never exposed to the user so this change is entirely internal.

.../engine/table/impl/updateby/rollingformulamulticolumn/RollingFormulaMultiColumnOperator.java

malhotrashivam · 2024-10-09T21:24:46Z

.../engine/table/impl/updateby/rollingformulamulticolumn/RollingFormulaMultiColumnOperator.java

+
+    private final String formula;
+    private final TableDefinition tableDef;
+    private final QueryCompilerRequestProcessor compilationProcessor;


I couldn't follow where the formula and compilationProcessor are being used.

They are used in RollingFormulaMultiColumnOperator#copy, but that method is used to support PartitionedTable proxy() calls and can be made more efficient (by supplying the compiled formula column, eg.).

lbooker42 · 2024-10-17T17:46:26Z

Python support for this is incomplete, need to update function signatures and documentation, plus include examples.
Have added this support in the latest commit.

Base/src/main/java/io/deephaven/base/ringbuffer/ObjectRingBuffer.java

rcaudy · 2024-10-18T20:13:20Z

engine/table/src/main/java/io/deephaven/engine/table/impl/updateby/UpdateByOperator.java

@@ -40,6 +40,8 @@
 */
 public abstract class UpdateByOperator {
    protected final MatchPair pair;
+
+    // Input columns for this operator. Must be dynamic to support discovery (e.g. formula columns).


You said this is OBE.

engine/table/src/main/java/io/deephaven/engine/table/impl/updateby/UpdateByOperator.java

engine/table/src/main/java/io/deephaven/engine/table/impl/updateby/UpdateByOperatorFactory.java

rcaudy

Looking pretty good. We can take a more thorough/complete pass next week if needed, but I don't want to duplicate effort too much. Do want to look at things after refactoring to rely more on SelectColumn and Selectable.

.../engine/table/impl/updateby/rollingformulamulticolumn/RollingFormulaMultiColumnOperator.java

…y value.

…entity values.

rcaudy

Just need to look at the operator code (base and multi). Deferring Python.

rcaudy · 2024-10-23T20:42:15Z

Base/src/main/java/io/deephaven/base/ringbuffer/CharRingBuffer.java

@@ -147,7 +155,7 @@ public boolean add(char e) {
     * @param count the minimum number of empty entries in the buffer after this call
     * @throws UnsupportedOperationException when {@code growable} is {@code false} and buffer is full
     */
-    public void ensureRemaining(int count) {
+    public void ensureRemaining(final int count) {


More opportunities to add @Override to this code.

rcaudy · 2024-10-23T20:44:28Z

Base/src/main/java/io/deephaven/base/ringbuffer/ObjectRingBuffer.java

    public void clear() {
        tail = head = 0;
+        // region object-bulk-clear
+        Arrays.fill(storage, null);


Is there are an argument for filling less, since we knew the head and tail?

rcaudy · 2024-10-23T21:54:00Z

...e/table/impl/updateby/rollingformulamulticolumn/windowconsumer/RingBufferWindowConsumer.java

+    /**
+     * Create a new {@link RingBufferWindowConsumer} for the given buffer.
+     *
+     * @param buffer the buffer to manage and


rcaudy · 2024-10-23T21:55:07Z

...e/table/impl/updateby/rollingformulamulticolumn/windowconsumer/RingBufferWindowConsumer.java

+    /**
+     * Push {@code count} values from the input chunk into the ring buffer, beginning at {@code index}.
+     *
+     * @param index the index to push the value


Suggested change

* @param index the index to push the value

* @param index the index of the first value to push

rcaudy · 2024-10-23T21:55:19Z

...e/table/impl/updateby/rollingformulamulticolumn/windowconsumer/RingBufferWindowConsumer.java

+     * Push {@code count} values from the input chunk into the ring buffer, beginning at {@code index}.
+     *
+     * @param index the index to push the value
+     * @param count the count of the value to push


Suggested change

* @param count the count of the value to push

* @param count the count of values to push

rcaudy · 2024-10-23T22:00:42Z

table-api/src/main/java/io/deephaven/api/updateby/UpdateByOperation.java

+     * reverse window, so calling this with {@code revTicks = 1} will simply return the current row. Specifying
+     * {@code revTicks = 10} will include the previous 9 rows to this one and this row for a total of 10 rows.
+     * <p>
+     * The provided {@code formula} should specify the output column name Some examples of formula are:


Apply to all similar methods.

Suggested change

* The provided {@code formula} should specify the output column name Some examples of formula are:

* The provided {@code formula} should specify the output column name. Some examples of formula are:

rcaudy · 2024-10-23T22:05:08Z

table-api/src/main/java/io/deephaven/api/updateby/spec/RollingFormulaSpec.java

@@ -87,11 +153,4 @@ final void checkFormula() {
            throw new IllegalArgumentException("formula must not be empty");
        }
    }
-
-    @Value.Check
-    final void checkParamToken() {


I wonder if we could do a Selectable.parse up front to determine whether the output column name is specified when param token is missing. Maybe redundant, but that's sort of in the spirit of our immutables conventions. Could also conceivably make that selectable() a lazily-computed, cached field.

rcaudy · 2024-10-23T22:07:48Z

engine/table/src/test/java/io/deephaven/engine/table/impl/updateby/TestRollingFormula.java

Please be sure you have:

Tests for 0, 1, 2, and 3+ inputs using the new expected syntax.

Tests that you break with >1 input if paramToken is supplied.

rcaudy · 2024-10-23T22:16:01Z

engine/table/src/main/java/io/deephaven/engine/table/impl/updateby/UpdateByOperatorFactory.java

+                vectorColumnNameMap = new HashMap<>();
+                columnDefinitionMap.forEach((key, value) -> {
+                    final ColumnDefinition<?> columnDef = ColumnDefinition.fromGenericType(
+                            key, VectorFactory.forElementType(value.getDataType()).vectorType());


Add component type (that is, value.getDataType()). I wonder if that means we need more testing for Object inputs.

rcaudy · 2024-10-23T22:20:11Z

engine/table/src/main/java/io/deephaven/engine/table/impl/updateby/UpdateByOperatorFactory.java

+            }
+
+            // Get the input column names and data types from the formula.
+            final String[] inputColumnNames =


You should validate that getColumnArrays() returns an empty list.

lbooker42 added 2 commits September 16, 2024 15:31

Refactor to allow dynamic affectingColumns in the base class.

265e334

Working, tests passing but need more.

5c5294f

lbooker42 added DocumentationNeeded ReleaseNotesNeeded Release notes are needed labels Sep 26, 2024

lbooker42 added this to the 0.37.0 milestone Sep 26, 2024

lbooker42 self-assigned this Sep 26, 2024

rcaudy requested review from rcaudy and malhotrashivam September 26, 2024 22:00

Added support for multiple formulas per call, spotless.

c8c87bd

malhotrashivam reviewed Oct 3, 2024

View reviewed changes

malhotrashivam reviewed Oct 4, 2024

View reviewed changes

Addressed PR comments

b64b342

lbooker42 changed the title ~~Add multi-column support to UpdateBy RollingFormula() operator~~ feat: add multi-column support to UpdateBy RollingFormula() operator Oct 9, 2024

Corrected JavaDoc failure in RingBufferWindowConsumer

66dff62

malhotrashivam reviewed Oct 9, 2024

View reviewed changes

Addressed PR comments and added python implementation.

eaa2816

lbooker42 requested review from chipkent and jmao-denver as code owners October 17, 2024 21:02

lbooker42 added 2 commits October 17, 2024 18:02

Updated python tests.

37298c4

Minor updates.

efc533c

rcaudy reviewed Oct 18, 2024

View reviewed changes

.../engine/table/impl/updateby/rollingformulamulticolumn/RollingFormulaMultiColumnOperator.java Outdated Show resolved Hide resolved

lbooker42 added 3 commits October 21, 2024 11:44

Refactor to use SelectColumn instead of FormulaColumn

c969c11

Aggregating ring buffer correction to reset cleared buffer to identit…

b3a02fb

…y value.

Change AggregatingRingBuffer code to be smarter about resetting to id…

36756da

…entity values.

lbooker42 force-pushed the lab-rolling-multi-formula branch from ef9aab7 to 36756da Compare October 23, 2024 21:40

deephaven deleted a comment from github-actions bot Oct 23, 2024

rcaudy reviewed Oct 23, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add multi-column support to `UpdateBy` `RollingFormula()` operator #6143

feat: add multi-column support to `UpdateBy` `RollingFormula()` operator #6143

lbooker42 commented Sep 26, 2024

malhotrashivam left a comment

malhotrashivam Oct 3, 2024

malhotrashivam Oct 9, 2024

lbooker42 Oct 17, 2024

malhotrashivam Oct 9, 2024

lbooker42 Oct 17, 2024

malhotrashivam Oct 9, 2024

lbooker42 Oct 17, 2024

lbooker42 commented Oct 17, 2024 •

edited

Loading

rcaudy Oct 18, 2024

rcaudy left a comment

rcaudy left a comment

rcaudy Oct 23, 2024

rcaudy Oct 23, 2024

rcaudy Oct 23, 2024

rcaudy Oct 23, 2024

rcaudy Oct 23, 2024

rcaudy Oct 23, 2024

rcaudy Oct 23, 2024

rcaudy Oct 23, 2024

rcaudy Oct 23, 2024

rcaudy Oct 23, 2024

		// noinspection unchecked
		result = new ObjectRingBufferVectorWrapper<T>((ObjectRingBuffer<T>) buffer, (Class<T>) componentType);

	* @param index the index to push the value
	* @param index the index of the first value to push

	* @param count the count of the value to push
	* @param count the count of values to push

	* The provided {@code formula} should specify the output column name Some examples of formula are:
	* The provided {@code formula} should specify the output column name. Some examples of formula are:

feat: add multi-column support to UpdateBy RollingFormula() operator #6143

Are you sure you want to change the base?

feat: add multi-column support to UpdateBy RollingFormula() operator #6143

Conversation

lbooker42 commented Sep 26, 2024

malhotrashivam left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lbooker42 commented Oct 17, 2024 • edited Loading

Choose a reason for hiding this comment

rcaudy left a comment

Choose a reason for hiding this comment

rcaudy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

feat: add multi-column support to `UpdateBy` `RollingFormula()` operator #6143

feat: add multi-column support to `UpdateBy` `RollingFormula()` operator #6143

lbooker42 commented Oct 17, 2024 •

edited

Loading